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. ABSTEiCT - ' . . 

. ^ Tb^ Instructional Strateg; Di^gifostic Profile (ISDP) 

vas designed to enable instructional developers and evaluatcrs to 

^predict the effectiveness^ of ^ -anjjh prescribe inproyeeents for existing 

^^^hstructional materials, ' Experinental 'studies iierV ccc^uct^ed In 
,introaactory college statistics and physics classes .to 'validate the 
ISQP and its accoipan;ing design prescriptions, Ivo lethcdclogies 

Qsed: (1) existing instructional (et^rials Here ftodified-^ bas^d 
on ft ffaiected prescri/tian r%sulting frci ISDP analyses of .those 
■aterial^r and tvo ofT mor% versions of th€ materials uer^ ccipared; 
and (2) a treak anit of an existing coerce vas identified and ttodifi^d 
-^iav se^el^^- prescrlplfions xesulting frco an iSDf analysis, lest 
perforiance* affect^ confideace# and tia€ v^re ccft^ared fcr students 
using the t^vised liaterfals* and students tising^the original . 
taterials, vh^J used to xevis^ exj,sting lat^rials^ th€ IspP 
prescriptions "^prodaced Significant diff^r^ifces crl; in-^th^ second 
study. Failure to oEtaii predicted revisits lay h^ve been *due to 
confounding factors in the experiaant-al situaticns, Othex studies 

,have demonstrated that existing materials revised according to ISDP 
prescriptions caR significantly iapron^^tadent perforianc«# 
especially if , the interaction vith th« lat^rials c^b te controlled 

^^and the tests can- be revised to sore adegnately leasure concept 
classification and rule^using behavior/ Uuthor/J&C) 
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'UNCLASSIEIED - 



consisted of course Intervention In vhlA a v^ak unit of an existing course 
was, id€ffttifled and modified via several prescriptions jr^Sultlng. from an ' , 
ISDP analysis. 'Test peirforinance^ affect^ confldenQ*e» and time were compared 
for students usln^ the revised material's and st(udents u^lng. the original 
materials.' ^ , ^ ^ 

■ ' ■ * . - 

When tised^to revise^, listing ipaterials^ the ISdF i>rescrli)tlonfi produced 
significant differences ot^y In the 2nd s^tudyf- FalXure" ta> find the predicted 
results may have 'bem a resttl^ of confounding factors In the^r^al-world « 
experimental ^tuatlons used. Otl^esji studies have demonsti'ated that existing - 
materials revlB«d according to IffDP^'prescriptlons can be demonstrated tp^ * . ^ 
produce <slgnlf leant Increases In student performance especially If t^e V 
Interaction with th^ materials can i)e controlled and th^ tests cati^be 
revised to more adequately measure concept classification and rule-i^lng 
behavior* " !^ " . / 1 



UNCUSSIFIED* 



^ StCUmTY CLAWFtCATIOM OF TmW WMtm^m DM» SntM«« 



FOREWORD 

XhlS; research and develofjment was 'conducted In support of. Advanced 
Develo|i^nt,: Subpfoject ZOIO&^SOA (Adaptive Eicperimental Approach to In- 
structional Design)* Thi's wotk is one aspect of ^n area concemed with 
evaluation of instruction/training. Previous', work reviewed the existing 
tes^axch lltetature that has^-investigated the propositions underlying the« 
Profile*^^Kfrhe review identified requirements for research and development 
to which the studies described In the- pre^seUt repott> which. Evolved test 
and evaluation of the ^SDP/w^r^ d^ec'ted-i/' ^ , '^^ 



- Drs..John Carter an^^Jphn £},lis served as contract monitots*^ This 
report was reviewed and edited by Dr* Joln\ Ellis; 



J. J. CLARKIN 
Commatifdlng Off leer 



'■ • . SUMMARY ' . ■ . 

Problem ^' ' *w ^ . , . 

The Instructional^ Strategy Diagnostic Profile (ISDP) was , i^ecdgned to ' . 
enai>le instructional/ develppe^s and, evaluators ta predict the effectiveness 
of and prescribe improvements for' existing instructional materials'*. Vhile 
some ^idence exists as to dts effectiveness » it ha^' not 'yet received su£^' 
ficient empirical eyalustion* - , 



Otflective 
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' The purpose of this research and development effort was' to further the 
development and validation of the ISDP Jth^ accompanying instructional* 
'design prescriptions,""^, ^ . 'V*' - <^ ^ ► ' . ' ; 

Approach ' ' ; ^ " , 

' ' ■ > ' ■ ' ' ' , 

Three empirical studies w^re cond^^ted using two different methodalogies; - \' 
Method one, vh^ch vas'used for the first tw$ stqdies, consists of modifying- 
existlsg Instructional materials based on prescriptions resulting from dn ^ - - . 
ISDP analysis of thbse oater^als. Method twa^ whicK was used f on^ tJie., tiiird 
study, is an intervention process. In which a. we^ unit of an -existing c^ur^e 
^ is selected modif iedVla'^several prescript ipna resulting from an ISDP \. 
^analysis** Test^ performance, affect,'*confideace; a'nd time were compar&d for' 
students using the^ revised 'materials and for those using the ofigiji^l materials^ 
^ ■ ■ ■ ■ , " ■ • 

Study 1, using, method one 0rt an introductory statistics' cDurs^^ compared 
a^'framewo^k presentation, with k regular prose presentation with either elaborated* 
or correct ^hsirer feedback* ■ " ' ' ^ ' 

Study 2, using t^thod one^dn the same statistics course as ^tudy i; tpm^)* 
, pared the performancS'of groups^ using connepted rul^s, discrete rules^ atPijd * ^ 
rules embedded in expository text* The connected rule involved an^ algorlj&hm 
for selecting which rule to use; the. discrjBte rule contained an algor£bl^; 

and the embedded ryle neither involved or contained an a],gorlthm* ^ > 

^ • . ' ■: r ' ' "V , 

Study 3, using method two, revised ^ unit of the s^llaj^us fo^r ;an<''irtt;roductory 
physics course and then compared that unit with the original; unit itr^ terms of 
Btudehtsperf ormance, time, and affect* Lectures, textbooks, and* ^(IIqcuss ions 
vere tite same for both groups* ' ^ . ' - < 

Findilngs , . ^ ■ t ^.^f, . . 

-~ ^ ' \ ^ 

In Sjiudy 1, no dlgnlf leant differences wete^ observed 'in d'iTudtht; perfo^^ - 
affect, confidence, or time* In Study 2, posttest p^erforiiuq^p&^o both discrete 
and connected rule groups was superior '^to. that of ' the- ^mi&^ded ^Id group. On 
affect, the discrete rule was- most positive,' ^ollpwed -by%\id connected and 
embedded rule groups. There were no time or cjopf £iS%nce^(fi|Pferences* Finally,, 
In Study 3« there were no Significant difj^erenaes 'liv^er^futiaance, time, or . 
affect between the two groups7 although the. means were '£ti the predicted direction* 
Failure to find the predicted results may tiave b^en'^a^;result of confounding 
factota in the real-wrld experimenltaL.s^tiuatidn^^i^e^*' 



'Conclusions ' * . ' 

~ * ' ■ ' ^- . 

* * The research reviewed Indicate that the propositions un4erlyi:ng the 
Profile seem to be VAlld^ While the data reported In this document Is ^ 
somewhat Inconclusive and not sufficient to make unqualified statements 
It Is^ 'nevertheless, positive* When considered with other data on the ISi)P, 
^It seems reasonable to assume that,'wh^n the ISDP, Profile 1^ used as a guide 
to analyze and modify existing Instruction, 'th^ resulting performance of ^ 
students Is likely to be more effective* This I9 esp^clallf likely when the 
tests OB well as the main line instruction ^caii be modified* It Is less 
likely when only the student syllabus Is modified* The ISD? does seem to 
have considerable potential as an Inatrdctlonal evaluation and development 
itool* , 

Recotnmendat io^ * 

^ 1. The ISDP, as presented In the ISHF^ training manual. Is recommended 

for use by Navy'-lnstructlonal developers and evaluators* However, It should 

be considered as -an experimental tool and should be used only by experienced 

instructional technologji^ft who call- appropriately adapt Its use to various 

settings and clrcTjmstatices* 

~ ^ »" * ■ » " 

s ' - 

\ '2. It -Is recommended th^t .ISDP validation and development efforts con- 
tinue so that this £hstrument can .become an easy to use tool for all instruc- 
tional development and evaluation personnel. . ' '•' 
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\ ' ' • ' ' INTRODUCTION ■ - , 

Problem . ^ ' - * * 

Gdidelii^es for predicting instruCtion^l^effectiVeness, ±i they exist '' 
at all, oare v^gue at best^ It is almost im^QBsible to look ht an instruc** 
tional- product and predict^ its^effectiv&ness by^the use of existing guides* 
The Instructional 'Strategy Diagnostic Profile (ISDP) was designed (Merrill 
(t Wood,^ 1975) to Enable instructional developers and evaluators to predict 
the effectiveness of and prescribe Istprovemetfts ^or existing instructional 
ma'terials* While ^oiofi evidence e^cists as to itd ef fectivene^a,H it has n9t , 
yet received sufficient ^empirical validation* 

"* ■ , /' ' ' ' • . ' 

Purpose . ' , * 

The purpose of this research and development effort was to further the 
developmant ai^d validation of thc^ ISDP and the accompanying instructional, 
' design prescriptions* ^ ^ 

Backgrbund and Scope . . ^ 

This prf>ject,was conducted in three phased* Phase I consisted of an 
extensive review of reported research studies as' they relate to' the pro^ 
positions underlying the Instructional Strategy Diagnostic Profile (Merrill, 
Olson, (t Coldeway, 1976)* Results, of ^thl^s review indicate that there is ^ 
considerable empirical .research' support fpr most <!xf the propositloiVB under- 
lying thfe Profile* . ^t was further suggested that an Instructional package , 
that' is judged to, have a high ISDP index shoulcf provide rather effective^ 
instruction* / . . 

This documep^ is the technical report for the'Phase II e^for,t, which, 
consisted'.of thi^ee empirical studies using two diffet;etit methodologies* ^' 

Phase III involved the preparation and validation of a manual for train 
users in ISDP analysis* ^^his manual will be pu1>lished as a separate tebhni 
report (Merrill, Wood; & Riqhards, in'|)reparation) * Preliminary validation 
indicates that experienced ins true ti^Q^l developers, who have alr^ad^^h^^ . 
graining in the vocabulary of the*t<SX^, are abU^^to^nsistently rate exis 
instruction and to prescribe mod indications by j||flng tlie guidelines provid 
by the ISDP\ttaining manual* * 



^ / ■ APPROACH * 

Afi' indicated^revtou6ly> Phase 'II this project consisted of conducting 
three empirical stvtSI^sr^sl^ tvo. different, methodologies* ^Method o^e,* Vhich 

jms -use^ for 'the- first' t^ stbdies, consists of modifying existing instructional 
materisls based ^ontprescriptions resulting ^from an tSDP anslysis of those 
materials, <^Mettiod tw{>, which wss used for the' third study, is sn iht^rventiop ; 
{)rqces8, in which (1)- a course is*snalyzed via the-ISDP, (2) a weak unit of. 

>inStrudtion is selected, (3> the t^st used fo^ that unit is revised such that 
it yields 6 higlier ISDP index; and (4) th^ unit of Instructioil selected^ is 
modified so thst the strategy used yields s^ higher ISDP index* The original 
and revised instructions sre. then sdminis^ered to rsndomXy^assigi^e'd^^^uups, 
snd performance on^^th is comps^red* Th^ three studies are ^ag ribed t^iriefly 
below an4#fin detail in the following ^^ectfifens* ^ * *^ , ^- ^ . " . - 

Study l"FramevDrk Rule Representation and Elsborated Feedback in Statistics 

. ^ Ihatruction, " ^ . ^ ^ 

■ * * ^ . ■- ■ 

y ^Using method one ih an introductory statistics course, a frsmework 
rule^representstion was compsred'with a^ regulsr prose rule representation 
snd' correct answer feedback was compsred with elsbo^^ated feej^b^ack*^ -i^On the 
Ijpsttest, there were no performancerdifferencesi.^ Students were also compared 
oh the appeal of the instruction, ^s measured by a questionnaire, (2) 
confidence in their responses, and (3) >tlme, required to complete the instruc* 
tioi^* There were no significsnt differences on sn^of ^hese dependent measures* 
The ISDP rating gf.the instruction before the mqdifidatioh. was very good* It 
was suggested that the relationship between performancei^d the .ISDP index is 
a decreaaitig function* That is, it requires a large increment st the^ high 
end of the ISDP , scale to result. in a measurable performance difference, ss * 
compared to s small increment st the low ^d of the scale* ^ 

Study 2"Test and Generality Consistency in*a Classification Tssk 

Also using ttiethod one- in the same Introductory statd^st^os course, 
perfonpance Qf, groups uslhg^ three types of ruie statements were compsredt 
conivBcted )rules, discrete rules, and the regular expository test, which * 
served as 'the control* The first t;^ treatments consisted of an integrated 
algorithmic floW chart represenj:stion ^for, the cannecj:ed condition and separate 
nonintegrated flow/chart representations for the discrete condition* The . 
regular condition did npt present a how^to-use-the-^rule algorithm of any kind* 
,On,postte0t performance the connected^^nd discrete rule groups scored signif* 
^icantly higher than .the regulsr instruction Tgroup* "^On, affect, all three- 
gtpups were^<{lffer^t with l^he' discrete gtpup most positive, the coi^rtected^ 
next, and^the' regular It^struction lesst positive* There were no significant 
differences between groups on j^o^ttest time or response confidence* It was 
concluded tha^ providing the student With sn aXgoritt^c rule resulted In s 
perfotmance increment* It should* be noted that this study also used materials 
which had s' high ipitisl ISDP rating and that the addition of sn algorithnclc 
repr^entation of the rule was st^ill able to-c&ude a further ificrement in ^ 
performance* , , ^ ' * . 



study 3"*Vall4atlDn of the Ipatructlonal Strategy Ptegnoatlc Profile 
In Physlcs^OO * * - ^ \ 

The seqoncl toe thodo logy was used in an Introductory phjrsips course 
at Brighsm Young University* An ISDP snalysis of the tests used in the, course 
^Indicated that less than 20 percent of the test items met the ISDP require- 
ments' fot aclequate rule^using* Perfotioance on the rule^using it^ms was 
signif icsntly lower than perforntance on the.memory-K>riented items* One^of 
the poorest (ss indicated" by test performance) units o£ Instruction vss 
selected for ISDP^ modification* A revided test was^prepsred which included 
more rule-using itemSj, snd the instruction was revised , to score higher on^^ 
the Profile* Dui^pg dhe sOmmer term,, students In the course were xhndomly 
^sssigned to the existing ojr^the revised materials* All students took*both 
the original Snd the revised test*' ■ ' ' - ' 

* *^ 

Th^re Va&a significant difference within groups, itidicsting t^st 
performance ^on encountered test items was better thsn that on unencountered 
test items^ However, between^^roup differences, on either type of test, 
while^invthe predicted direction, failed to reach significance* ^The ^tudy 
^sttempted to demonstrate thst modificstion Qf the syllsbus in atcordsnce witti 
*ISDP T>rinciples would ifesult in 's performance Increingnt* Betfsyse of consider*- 
sble within group vsriance probably ^resulting 'from the uncontrolled influence 
of lectures, the jtextbobk/ snd student intersction, the results failed to 
demonstrste the predicted ^difference* ^ - . . % 




STUDY 1-: FHAM^OEK RULE R^RES^tlTATrON-Alto 
ELABORATED PEEDBACK IN STATISTICS" 
INSTRUCTION^ 



Design Challenges , • ^ ■ 

' The development ol Instruction that effectively ptfepatea . learners to- 
ase complex rules un^er low prompt testing conditions presents two chal** 
lenges to the instructional Heslgner; ' (1) finding an appro'prlate represent- 
station of the ruXe(s) that facilitates recall, and (2); providing for ' 
appropriate practice and feedback that effectively prepares the learner to; 
apply the rules on similar test items. " ^ t 

Rule Representation / ^ ^ 



V Several sources provide evldence for the necessity' of repi^esenting 
rulea in Instfuptlon vlttr accompanying ntathemageniif inforpatloti; ' Landa 
(1974) ,found that thk effectiveness and efficiency of studend perfonoance 
inc^reased when algorithmic representation was us^d in teaching mathematics . 
and language rul^s^ Markle (1975) found that a vertical list of attributes 
was superior to^-paraeraph with embedded attributes in the acquisition of ^ 
rules. May|^r (1975a) cbncluded th^t an assimilative set or framework 
facilitated^ storage and retrieval of rules^ f rbm memory^ Minsky (197^ sugr 
gests that a framework of Infarmatlon facilitates the inanipulatl&n of critical 
jBleioei^ when encountering a new situation or Instance. Closer (1976), 
advocates finding ways to represent cou^lex information to the novice learner 
in such, waya that encoding is facilitated and time to criterion competency is 
decreased. ' . ' ■ . * ' ■ 



Appropriate Practice 



- Practice can be defined as an instructional display that (1) requires 
the learner to respond overtly to an explicitly stated task and (2) provides/ 
at* least; correct answer feedback to the leamet* Practice is judged' to be 
appropriate if the task, co;ntent»^and feedback of the Insttuptlonal display 
are isomorphic to the task and content of the' rule representation, is 
assumed in thl^ study that t^e most effective practice displays should include 
-framework displays identical to the Tule representation^ and that feedback t 
should be the correct answer with elaboration rather than the correct answer 
only* Merrill an4 WooU. (1975)^^ Wood» Richards^ and Merrill (1976)^ and ^, ' 
Schmidt^ Wood» and. Merrill '(1976). provide a rationale* and some evidence for 
creating, exan^le and practice displays that are isomorphic to or conslst^t 
with rule displays. 



^Study conducted by Uk- D..Wood, R. ^. Gil^trop, and M. D. Merrill. , / 

^A more .extensive definition of the terms used in this section la found 
injMerrill anq Mood (1975). Space limitations do not allow for a £ull 



elaboration here. 



^ Hypo Cites es ' ^ ■ ' " 

^ ■ -llri^ study. wl^l investigate^ the effecta of (1) representing complex, 
■ rui*fes vifthln a mathem^genic^ fraiileworfe pf Information and X2) providing 
elaborated feedback c^^^^ract^ce displays* The general hypothesis Is as 
foi^-ows^ ■ ^ , * * ■ 

^ ^ The fraajework '(mathem^genlc) representation of rules 
antf'ccSnslstei\t. practice with eXabor^ted feec^ack will, 
pr^>]uce significantly. nioT^ positive student outcomes than 
. s^ra^ght ' lis t^or^nonframework" representation- of rules 
wljth <?qfrrect answer only feedback-^ \ . 

Methods " *- ■ ' ^ 

Selection gf Subject .Matter / 

^ ^ GlaseW aa^ Resnl^k' (1972) have. Identified the need to do instruc* 
^ tlonal psychelol^ research wlthlQ realistic setting^ with existing curricula 
Basest on this f^AScqeiv^d* heed to go beyond the artificial context and subject 
matter of "the laboratory setting, "an ongoing Introductory statls^tlcs course"^ 
at Brigham Young University' wds seliected for the^followlng-feasot^s: (1) 
. typical subject .matter within a typical instructional setting w^s a^llable» 
' and C2) coniplesG subject matter conducive to framewprk rule representation 
was an Integrftl^part of th^ course, ' 

A se^tnenll^or -Wesson of Instruction i from fi unit of hypothesis tes&ing 
for one mean walSf^hosen as* the specific subject matter of the study* This 
segment^ Included ''the following six steps: ^ 

' ' 1. Forimila'te the null {H ) and alternative (H ) hypotheses, 
, * * * , ^ " V - ^ .a "^'^ 

V-"^ ' 2, Choose sample size (n) and alpha. ' ' , * 

\ - > A ■ . ■ ' 

3< Choose test statistic, ' - ^ 

^^4, Make decision ride, ^ ■ 

, 5. Calcujjafjte test 'statistic, 

6* HaVa^t i]^cislon, ^ 



Subjects 



A grpup of'^93 students (encouraged by the statistics department to 
participate^ wa^s^ glyon course credit; for participation In thf experiment, 
Most-^ the students ^were spphotnores 'and juniors, with a few seniors and 
graduatU &tudentk« ' A wide -diversity of majors was represented by the gtoup. 



' . / : ^The^ £our.typ«s of tr§iatment inaterials wer6: ^ ^ , 

1* Ftaniework rule representation wlt^ elaborated feedback*./,. ^« . ^ 

^ • — \. . . - ^ * ^^A'^ ■ - ^ '-''^ ■ 

2* ^ Framework rule repitesentatlon with correct answer feedbaejc* 

' V ^ ^ ' , \ * ' , ^, - ^ ^ ■ ; 

* ' '3*** Ronframework rule^represefttatloti with»;elaboratea feedback* i\ - 

^> ' \ 1. V* * V ^ 

4* , Nonf ramevork rule^ represent^^tion with correct answer feedback; « ^ 

-■ ^ All four treatment material types were i^r workbook forro and were ^ i 
randomly diatributed to students in the lecture hHll:^ere t,he experiment' 
was conducted* . i ^ * 

Representative 'example&r of the rules^ practice^ and feedback displays 
used in the four treatments ^afe found in Figures 1 through Figures 1 
2 show the framework rule' representation with elaborated and with correct;* C 
toswer feedback^ respectively! and Figures 3 and 4; the nonf ramework nile^" . 
represexitatlon with elaborated^^nrd with'^doiftect answer feedback*' 

The treatment condltioS^' shown, in Figure 4 represents the of^gin^ 
instructional materials, used in. the course* These msiterials appeared in 
the form of a self -ins ttjuctionei t'e^^ by Christensen (1974)* A& agreement'- 
was-'made wit|> the instructo|^ th^t' any . treatment that was considercfd ^y means 
of the. Instructional ^Strategy Diagnostic Profile (ISDF) (Merrill & Wood» 1975) 
as less effective than the'(^7lginal would noit. be used* Therefore^ the £reat«^. 
meats shown in Figures 2 and 3 were consid^ed to Ve more effective^ whl^e / 
the treatment' *in Figure 1 was considered to be the most effective*.* \. ^ 

^ - Instrumagt^tlon ' „ * / ' ^ ./ 

The post-treatment tes$ administered to all /^stu^^n^ in the experiment 
comprised 12 paper-arid-pencil> multiple-choiqe quertions of>recall> concept 
claaslf ication^* and. rule-^sing typi£s* An example' of a rjule-uslng test item 
is found in Figure 5/' Tti)^ nOmber of correct anaweits on ^the twlve items by 
each student was.uaed as the dependent variable performance* 

G^h jquestion^as followed by a aeven-point.*dif f ereptial sc^le^ which 
probed .the studentj^ confidence in his answer to^^he test ^question* An example 
of the confidence scale itemMs shown in Figur^ 5# The average icdnfidence fot 
each student, on all items wa^ used as, the dependent variable conf idence ^ 

i . The affect^ that;,, the Instruction Kad on t^he student was measured 'five 
t|pes during the treatment peViod» Using a aevtoT<|uestion semantic differential 
scdfe fpniajt^Caee Figure 5)* The meases were^aken after students (1) read 
the inftructional materials* (2) respmded to. test l^ems 1 and 2 (meouitry type)» 
(3) responded ito teat items 3» ^4» 5* and 6 (concept and rule usiftg)^' (4), ^ 
teapbnded to tetft ititos 7* 8» 9» and 10 (concept and rule.usipg)^ and (5) ' . 
responded to ^ test items 11 and 12^ (higher order rule using)* The word^irs/" 
were scTanbled as to order and polafrity for the five measures in orderj||^> 
avoid, an 'anticipat£lon'ef£ect« /llie a^^erage score fot each Btiudent ^n £dl five 
Affect' erasures waa used as theMependept variable affect* 



Tlie six" steps 



of this procedure can be remembered mote ejtsily if they\ 



are. listed in a framework similar to- the one below/ , 

^ - ^ — \ • , } 



I'- 



i ' ^ I^6riV|ULAT6 Hp AND, Hg' 



2^;^^ CHOOiS^^t* 



a. CHOQSE TEST STATISTIC 



4/ MAKE DECISION ftUL£ 




5. CALCULATE TEST St ATISTIC 



6; MAKE D^'CIStON 



y th(Js^^,5ix steps/Will be discussed in more detail within the above fraine- 

The examples for illustra"tion,.3nd the practice examples will be pre* 
swrteff using this framewbrbdn order to assist in learning arid remembering^ 
the 6 basic component;? of hypothesis testing. , . * 



Fl^rje 1| Framework rule representation with elaborated feedbacjt; , 



% 



PRACTICE 



In the followlag: probletiis,' use the 8lx-»6tep procedure discussed In thls^ 
section to test the hypothesis for means of normally^^dlstrlbuted pppulatlo'nd , <^ 



when 0^ is hot Icnown'; 



1^ vThe follawlng are measurements of Brix degree^ On Jtnolasaes; 82*0, 79i^6» 
78/4, 61.JB, 82*2, 79*9, 83*2 79*9, 82*3, 84*1* In order -to.be graded as high 
quallt^y molasa^a, the Brlx degrees muat be equal to 80* At an d value of 
0*05, could the molasses from which the s^ple^ were taken be gi^aded as high 
' quality? For thla data, *± - 81*34* s * 1*8, ,an<J AO - 3*16* 



' 1 . • ■ 

* 


\ 






J 

/ 


2. 

\ 




3. 




1 

i 



4. 



5. 



6. 



^ / 



Figure X« (Continued) < 
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Answei's to 
"Practice Problem 4 



a CHQOsli^oe 

'is*;" - - , 

-n ="10. . 



3: . CHOOSE TEST SlyCriSTIC 



X ')V|/^ DlciSlOtg RULE 




-V. ^ Kcgiwt . 



"is,- 



Rejcttaon 
Region ' 



5, CALqOLATE TEST STATISTIC. ^ 



tr-f*--' 1 — = Z.354 



6^? MAKE DECISION 



SSftte t >,,to25- C=^2,354 > 2.26'i) 



^, we rej(ict the H . 



S 



0 



Ren^&nber^ that* a statement of" 
equality- (either or -Jwill ^ 

Tdi^ays appear in the Hq.' An « / 
Sign in tHI always defines a^ ■ »^ 
twp-tg^^led test. The Hg^ is always 
tife cqfl^lement of the }t^'. 



*In Step 2, a is give/, and n is 
obtained -^y. counting the numbef of 
observations. ^ 

In Step 3,, = 80 ;md n = 10 are 
substi^ljted into the test statistic 

^ - -y^j^ 

The I ft (2.262) was obtained as 
follows: note: jtf stands for 
the .absolute value of t. 
at For a two-tailed *test divide- 
a by 2 (a/2 = ,025) 
^"''b." df (degrees of freedom) = 
n-1 = 9 
c. Obt^int-value from'table 
. ^using df = 9^and^ a = . 025 * 



Ij0b1(ing.at the fonmila from Step 3, 
we *>ee that the values for x and ' 
s arc missing. Using: the values 
given in the problem, ( x = 81.34 
^^nd s = 1.8) we calculate the 
^1:-value^ - _ J. 



LboTcing at the diagram in Step 4^ 
we see^thaf since 2.354 >,2.262, 
we are in th^ rejection region of 
the right-hand XaM- We ctixi Con- 
clude, thcrforc, that the molasses 
cannot bq .graded as high quality. 



^: Figure It XCorttlnued), 



The six steps of this procedure c§jqt remembered more easily If l^hey 
are listed In a framework slmllar^ to t^e one belov.r 



1. FOEMOLATE ft AND/S'C* ^ - ^ ' 

o \ a^if • ; ^ 

/ * 

* * J.. ^ 


2. CHOOSE n, « / 

t 


3. CHOOSE' TEST STATISTIC 

• r 


4. U&KE DECISION iim,E 

1 


1 


t / 






* 




5. CALCULATE TEST STATISTIC . ' ^ ^ 

<_ 


6. MAKE DECISION 



These six steps will be dlscusE^e^ in i^re detail within the above frame- 
works The examples' |or IXlustratl&n and ttie practice. exampjXes will be pre* 
santedjaalng this fir^mework in order to Assist in learning and remembering 
the 6 Saslc cdqtponents of l£yp9;thesis testitkg. \s 



^flgptre 2^ Framework rule ^representation with correct answer feedback. 



PRACTICE \ . ■ • 

- In the following ftroMems-, use the 6-step procedure discussed in this 
.^ecrion to test'th^ hypothesis" foy^means of nomialXy distributed populations 
i|bea 'Oj^ is„not Jcnown^ " ' t^i' 

/T\ ' f £ ■ ' - ' 
[IJ The following 3re tneilSUr^erit^ nf RriY deo-reec nn mnToccoc • 00 fi tn e. 

78.4j »1,8, 82.2, 79.^9, 83.2, 79.^, 82.3, 84.1. In order to be graded as high 
quality molS^Bses, the^Brix degree^ must be equal to 80. At^dn o value of 6, OS, 
could the molasses from which th$ Samples were taken be graded as high quality' 
For this data, X = 81.34,. s = l.,8,' arid 3.16. 




o 

i 
* 


^ t ^ 
* 


£** 

'^^ 

f 

t 


■ . . 


1 

1 


4. ^ 
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JPlgure (Continued) . 



Anw^XB to Practice Problems 



Lesson 2 Section 2 



■I. : 



4-- 



.1. (1) H^: V 7 8,0,< '"^a* ^ ^ ®° 



(3) t 



8- 7 -no ' , , 



-(4) Reject:.a>- if |t]j-^a^26?, otherwise accept H (note |tl stands for 
thft-^^soSuLtej^iue of-t>. i, ^-^ — 



(5) X -,81.34 t - 2.35A. ' 
* (6) Since 2.354 ~> 2.262 reject B^. 



\ 



Figure 2. (Continue<Q. 



2? 



13 



. iDEFINITfON ; ^ \ . / . 

A .Test for One Mean When o Is Not Known , A test of hypothesis for 

^ . ^ — : 

ojie mean when o is not known is a st^istical procedure used to decideavhe- 
thcr or not th^ mean of a normally distributed population takes on the value 
of iJ * This procedure differs^from-the one set forth in the^pj^cvious. section 



"Hrrtie^tc^rsTatisTic used andTin the decision .i-ules employed. In this sectioff^S 
is uJSGd as an estimator of o^* The 6 steps, j?f the procedure are as follow^;^, 

K Formulate H and H , The 3 possible hypotheses for the mean 
of abnormally distributed population" when d is not.known are: 

b. H : iJ' ^ IJ vs' H : iJ < p * <^ 

c- H : p = iJ vs, H ; p f p . 

\^ 0 0 a 0 ^ ^ 

2* Choose a sample si2e# n# and ^ value for a. ' , 

3. IiGt the test statistic be , , 

X - V " ' ' 

f= £ . . ... . - " 

s//n" 4 , ' 

* ■ 

4* On the basis of the a value , choose the (decision rule ^according^, 
to the decision i;ule table, table 17, 

5* Ttike the sample, and compute the test statistic. 

6. Apply the decision rule, and maKe the decision. \ - ^ 

." • : ' ' ' . '\ . ■ . . 



\ 

Figure 3* Nonf ramework rule repreaentatlop with elaborated feedback* 



PRACTICE- 

In the following probleoSt use the six-step^ procedure discussed In this ' 
section 'tQ test the hypothesis for, neens of nomally distributed populatlbns 
when ff^ Is not known: ' * ' . ^ * 

• • ' ' ■ ■ ■ ' ' ' ■ ! 

1. The following are measurements of Brix degrees on molasses: 82.0, 7916, 

78.4, SI. 8, 82.2, 79.9, 83.2 79. 9> 82.3, 84.1. In ord'er to be graded as high 

quality molasses, the^Srl:^ degrees must be equal to 80. At an*a value of 

0.05, conld the molasses from whlchr the sarople8^ere_ta ken .be graded as high. 

-tpirlity? — ^For-thl«~data;^^81T347~s~-"rr8, and *^I0 -~3..16. '' . ' 



' CD 



C2) 



C3) 



(4) 



C5) . 



C6) 



/ 



/ 



Figure 3. (Continued). 



\ 



■■■'ir 



-ft- 



(2) a - .Oy i^V loT 

(3) ' ^^ f^' ^ 



V Answers (d : 
. : ^ m. ' Pf^^^jtice Problem 1 

P jff 80 Rcme^Her that a'statcincnt of equality' ■ ' . 
. t^Wierj^ij>_r or^ always appear ' * 

'in'ythe ^^. An = s^gn in tKe, ttj always ^ 
■ . drtiites a two-taiXed" test. The is . ^ 
always the ccmpl^jneht of the Htj. - 

■' /■■. .' ' . ■ ■ ' " ■ ■ ' 
a is given and n is obtained by counting 

JXhe number ,of <*servatioij^. ■ ' . 

^fxip = 80 and n - 10 are subst^ltuted into 
,^-tne- test statistic £: - VoS 



<4)- RejieJ'fto if lt| > 
2.?fr?,» otherwise 
' adiefpt .a, (note: tl 
~ ^j§is for the al!-^ 
Spltfte value of t) 

w- ■ • 



The 



ftj (2.262) 



was obtained. as follows: 



a. For .a two-tailed test divide 
> . ^ by 2 Co/2 = .025) 

hi df (degrees of freedom) ^ n-1 = 9 

c. Obtain t-value from table using , 
df = 9 and ,0 « .025 




• Reject j«n 

Region 
(t<-2':262) 



Acceptance 
.^Region 



Rejection 
Region 
Ct>2.262) 



JS) X = 81.54 t =■ 2.3S4 




To obtain the calaiTofed t-value, it is 

necessary to look yt the fonnula fronj 

Step 3 (^-^"^^ ^j^g ygjjjgg ^ 

X .arid s that were given in. the problan, , 
we substittlte and compute as follo^^s:' 
81.34 - 80.00' ^ 

•TXTvnr ■ ■ ' > 



* t = 



4^^^^ 2.354 



(6) Since 2.354 > 2.262 
reject Ho 



Looking. at the diagram, in Step 4,, 
we see that since 2i3S4 > 2.262» we are 
in the rejection region of .the right hand 
tnil. We can conclude J thorcrore; th;it"the 
molasses caxuiot be aT'adcd as^ high quality. 



Figure 3» (Continued) > 
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.DEPlNltCOK 



: 'A Test f c>r Oite Itean.When ^? ik Hot Known. A tfeat of hypothesis for * . 
" , ■ ^ . X ^ _ 

one acan When Is not loto^wn is s ststistical procedure Used to decide whe- 

^tU^r or not the'mean of s nonnal^y distributed ^pulatlon tskes on the value ^ 

of u A This procedures differs from the one set forth in the previous section 

in ttte te^^ststistlc used snd In the decision rules employed. In this section 

*' ' ' . r / ■ , . ' ' . ^ ' 

0 is .used ss an-estiinatar o^^o^.'Hie six steps of the procedure are as follows: 



1. 



Fomilste H and E-. The three possible hyi^heses for the mean 
of a fiomaUy distributed population wJli^ Icnown arc:* 

,a. H : ' V < vsl -B^: v > u . 
0 '^0 a ^0 

b. -'H : u > u vs. H r v < v 
^ 0 a * .0 



0 

c. H : 



vs. H^: V 




2. Choose a zsmpXe size, n» and a value for a.^ , 

3. Let th0 test statistic be 



t - 



\ Oorthe basis of the^a value, choose the decision rule according 
to the d^ecision rul<f table, table 17. ■ , ^ ^ 

5. Take the sample, and cootpute the test statistit:^ 

6. Apply the decision: rule^ an4 make fhe decision. 



T 



Figure 4. • Ntrnframework-mile representation. with correct answer feedback. 
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, PHACTiCE 



\ - 



, In the following problems, use the 6-step proceciure discussed in this* 
;pection to 'test -the hypothesis for means of rtotuaUy^jdistributcd populations ■ 
wKen ffy! is not loiowh: ' ' . ' , ' 

" '■■ '. • . • " ■ • ■ " 

1,' The following are measuremenis of Bpjc^/icgrees on molasses: 82.0, 79.6, " 
78.4, 81.8,. 82.2, 79.9, 83,2, 7^9, 82,3, '84.1. In order to be gVaded jTs high 
quality" jnblasses, thfe Brix degrees must bettequdl to 80. At an, a value of 0.05,' 
could the jFiolasses from which th^ samroies Were taken be graded' as high quality? 
For this, data, x = 81.34, s = 1.8, and Vifo = 3,16. . * \ 



^1) 



(2) 



(3) 



(5) 



(6) 



Figure 4> (Continued). 



Ans«F«i^8 .to, Trtctlc* Problems 



Lwon 2- Stctioib 2 



'i. -ay . H I 80 ■ H : li il 8a 

, (2X. a - .05, n" - . ' 

X - 80 - ^ 

8 /, /io 




(3> t 



(4) Ri^ect H if Itl 2i2.262> otherwise adctept fi (itete it] stands for 
th« Absolute vaiue of t). . . , 

* C5) X- 8i.34 .t- 2.354 - ' 

(6) ' ^nc4 2.354 > 2.262 reject H^., ^ - 



/ 




Fi^re 4* XContintted)< 



i; 



i9 



90 



The effect workers using new tools 'on the number of clrcalt 
boards 3!^seaible(^ in an electronics plant is being tested. A r * 

^r3nd(^«sdmpl&Hof individual workers' production totals is taken: 

* '. * ' , . ' ~ ' 

- I, 3, 4,^ 6, 5i^5,c7, 8, 2, 2, 4,. 5, 7,^3, -6, 4. The average " 

\ ' " ^ . 

number^of citcutt. boards ^ol^ this sample is 4.5 with a standard . . 

deviations of . 2. The plant otanager wants to know if the sample 

average t)f 4.5 Is statistically different fromr the previous , average 



of 5,5. Let a = .05 

(Use this *area for 
Vfirk space.) ^ 



(Hark an X in the 'box that corresponds 
to the. best' answer for. each test item 
below.^ " ^ 



7. The appropriate formulation of 

^nd'U for thevabove problem is: 
o a 



a. II 



b. H^-: M = 5.5 vs, H^: ii ji' 5.5 



t: 

d. 

£. - H 



M .< 5. 5 VS. 'H,: (1 > 5.5 



V < 4.-5 vs. H^: \i ^ 



»i > ^.5 VS. H^: v < 4.5 



(1 7 4.5 vs. H v y 4.S»' 
(1 > 5:5'vs. H : (i < 5.5 



— — "^^ ^ : 

How confident ar£i you itl your answer to the a^ove question? 

t " 



Very, . 
confident 




Not at all 
confident 



Figure 5^ , Dependent variable mc^asures including an example of a 
. , ' rule-using test item," confidence scale, a semantic 

, ^ ' differential affect scale, and a time record space* , 
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ijiterestins : . : : i' : 


: botfn» 




worthless : : v i : 


_j beneficial 


coippleteu L ■ ' ' : = = 


: IncOnfii^ete ^ 


detestable ^ i : : : : 


' ' enjoyable ^ 


cleifcr : . : : : : 


: - confusing; 


relevant : : / : : f ^ 


' : irrelevant 


redundant ^ : s r • : . : 


, : ^ concise 










Please record the time on the clock: 
1 













^Figure 5* (Continued), 
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^ Mea^uris of the amount of time spent by individual students on 
sections of the treatment materials wer^ taken at the same five points 
as th§* affect measures* Each ©fudent .wrQte"Tt6wB' the time from the wall 
clock in the space provided (see Figure 5)* The total- elapsed time taken 
dueling the treatment p^rio^d hy each student was used as . the dependent 
variable time* ' ' . 

♦ ' ' ■ 

- A 2 X 2 factorial design with'a multivariate analysis/of variance 
(ANOVA) was used to assess the effects of treatments across all four 
dependent variables/ A generalized AMOVA program which adjusted for unequal 
cell sizes (Bryce.£t Carter, 1974) was used to analyze the datal 

- • \ ' : .- . ' ^ 

Results ^ V ' ' . ^ J ' 

The following fbut hypotheses wei:e tested: t 

1. Hypothesis 1 ' ^ ^ 

Performance scores* will. be", higher for the framewE>Tk rule /elaborated 
feedback traatmegit group than fbr the nonframework/corregt answer treatment 
group* ' ^ ^ ^ - t ' ^ ' ' ^ 

. \ - . ^ 

2, Hypofcesis 2 

dontidehce scores will be higher for the framework rule/elaborated ^ 
Feediydek treatinent group than for the nonCeedback/correct answer treatment* 
groups 

3* Hypothesis 3 ^ ^ 

Affect scores will be higher for th&. f rameVork rylfe/elaborated 
feedback tireatraent ^roup than for the nonfraaiework/corract answer trreatmerit 
group. ' , ^ 

4. H^othesis 4 i 

- - ^ ■ . - ■ r \ 

To6al elapsed ti^ne will \>e less for the framevork rul^'/^^Ia bora ted 
f»ed&ack treatment group than for the nonframework/ correct "answer feedback 
treatment group* _ - ■ ^ 

*< 

A mulitlvari'^te analysis of variance (MAHOVA) was performed simultaneously 
on all four dependent varj^ables (performance, -conf idencft,^aff'ect, time) as. a 

'control for^ an increase in T^^pe I error through repeated univariate testSi. 
The MAKOVA F-test for the full 2x2 factorial model was not significant: 
F (4,86l) 0*697^ £ > *05- ^The means and standard deviations are reported in 

,T^ble 1, and the ^respective tF ratios, on Table 2* 



' , ' ' Table 1 - . ' / 

Meana and Stattdard Deviations of Deji^endent Variables for Four Treatment Groups 



Treatmaiit Group 



Dependent Variable 
Performance Affect ConfiSfence ■ . 

Heajx Std. Dev* Mean Std* DeV. Mean . Std. Dev., 



Time 
Mean Std. Dev. 



Framework Ruld 'Representation:. <; . ^ ^ , 

Elaborated Feedback 7.25 ' .47 5.26 .18 . 4.95* ; .llS^ 71.21 2.89 

• Correct Anpwer Feedback. a^2 ^ .48 -5.2i .19 5.05 .16 ^ 72i04" ' ^2.95^ 

Konframework Repreaentajtion; ' i - , . 

• Elaborated Feedback 7.39''' .48 4.8^ .19 4.81 ,.16 ^ v76-.09 i.95 

• Correct Answer Feedback 7.57 .48 5^41 ' . ,.19 4.86 . .16 - 74.87 2.95 



Table 2 

^Summafy of Univariate; P-Racios on Eour 
Dependent Variables 





* m 


* 


Dependent 


Variable 


- 1 


^ Source of 
Variation - t 


Ferfomiance 


Confidence 


- Affect 


■ Time- ' 


Rule Representation 


.26 

t 


l/06 ' 


.25 


• 1.73 


Feedback 




1.44 


.23 


1.64, 


i003 


K X f ' 






-03 , 


2.34 


.^22 


Hote* 


All F-ratibs 


were based on df 


= 1,89 and a 


= ,03.'. 
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Discussion 
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The mean scores on performance, confidence, and affect, as well as the^ 
total elapsed time for treatment, were chosen as the level of measurement,' 
^Qince more precise analysis (breaking each variable out intg, smaller cate^ 
gories) yielded no additional information* 



* The consistent lack of significant differences across the design in the 
study may be due to one or more of the following reasons^ 

1* The'^original versiorTS^f^the instruction (Figure 4) was considered, 
by means of an ISDP analysis, to.be superior to any other available i^riiitedr 
format material "on the subject* It is assumed that a less' effective treat^ 
ment (e.g*, embedded rules^in text, partial^r qo procedur^^ helps for using 
the rule) would have assisted in creating tliffexenceS between groups;, in 
ot'her words, by definition, the treatments were^Very similar* 

'^2* The 2 hours alloyed to tt^e experimenters for the treatuvent period 
was jiidged^to be insufficient foi^ the- complexity of ^ the subject matter involved* 
The amount of information to be processed was probably too much fo):4|£udents, 
regardl_ess of^ the treatment 'condition* It is assumed that the net effect of 
this time constraint drastically reduced Che betweeh-group variance that other- 
wise might Iiave existed* , - * ' ' 

Additional research efforts migkt (1) create greater differences in treat- * 
ments by embedding (or making less m^themagenic) critical attributes, and (2) 
allow for more time on task to determine if between-group variance can be 
increased* . ' 

If, complex rules can be represented to the learner^in ways that will 
develop skills of competent recall and use^ (application) in realistic test 
situations, a valuable tool for^the^ ^-nstructional developer to increase 
the effectiveness and reduce tlie cost of instruction could be made available* _ 



. • STUDy,2:''^^n^^.'AKD GENERALITY "consistency IN A . , - . 

: .' ',■ STATiS.TIpS C^SS'm.fcATIQN TASK^ ; , * I 

, ' . , \ _ ' . ' 

' Problem ' ■ ' > — * 

■ ^ / ^--'K?- 

Analyses, of testd oftei:r indicate ^<:requit)^d .test performance that is * 
^fiot^ entirely consistent wltti the aa^ociated instruction* ' the assumption 
of this study is that-instruction^should present the student with both the 
content of and, the behavior require^ for performance on a subsequent te^t. 

One of the tionrponentr of test -instruct ion consistency is a congruence 
bet^teert ,tjie test and the generality (statement pf rule, definition, or pro- 
p6sition\upon^hich 'the%nstructi6n is centered)* Thou gh some evidences * , \ 
exists to^^ljtdicate that a generality im^cts positively upqn p^erformance 
XMerrli^, Ol^en, i Coldeway, 1976), ^he effect of test-generality- isomophism 
has appsrentiy not ,beeiS[ specif icCLly tested* 

long ^is)e ago. Yum C1931) found that a slight change in stimulus ^ 
properties from instruption to test resulted in a significant decremeujt^^ 
successful responses on test performance* I^searchers have been slow ±£'^^'^ 
extending this sort of tightly controlled paiVed-a63oclate study into t^e 
mote complex .levels of instructional application (Glaser & Resnick, 1972) * 
At least part of the reason for this slow pace was sunmiariaed by Stkke Xl973), 
who stated ifKat neither scales nor grounds have be^u, developed for describing 
test and ii^struction similarity,^ though he cites some progres/ being made 
(e,gl, Anderson, Coldburg, £t Hidde, 1971)* Anderson (1972) recognizes the 
problem in a d^^fferent va^ when^he suggests that achievement tests are based 
on "things'" not clearly and consistently defined* Cropper (1970) has made 
some inroads, indicating an influence of spatial organization of materials 
upon st;udent response* Hayer (1975b) has noted a forward processing effect 
that shows a relationahip between the kind of stimulus materials used in 
instruction and the test response, 

Scandura*s use of the algprithm and higher" and lower-^rder rules in 
Instruction (ghrenpiels & Scandura^ 1974; Sdandura^ 1970, 1973, 1974) stresses 
the importance of specifying the precise behaviors requested of the learner* 
Shoemaker (1975) echoes this when he speaks for having identical elements in 
both instruction and test items* Cropper (1976) takes the position that task 
and content pos t 'Ins true tion^l test ai^lysis should include the same taxonomic 
categories as the ^'^f roht-end" Instruction to effectively diagnose learning 
failures. ^ 

Landa (19743 concludes that .^udents h^ve difficulties in' solving im^- 
countered examples because th6 general rules necessary for identifying specific 
solution rules are unidentified ahd not taught^. When Xhls Inconsistency is 
resolved, the intjbgratioa of separate rules is facilitated, and errors decrease 
rapidly over a relatively short period of s]^l)seqiient instruction* - 



^Study conducted- by R. V. Scluqldt, N. D. Wood, and H. D. ^r,rilii 
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^ All the forementlondd studies point to ,5 felt' need— and some evidence — 
that wlfat Is tested should have been presented previously .. to the student 
j^though th^ specif Ic Instance^ should differ). Just how c\Lose this match 
should be Is. opeiT'^d' question. A study by Scandura and, Durnln (1968), 
Indicates that a mlnpr shl^ away f rom' teet-generallty Isomorphism Is per- 
haps desirable. ' , ^ \ ■ .1 ^ 

Merrill and;'Woo<^(1974;/l9752, In their Instructional Strkiegy Dldenostlc 
Profile- (X^DP>, have taken up Stake's challenge. ^ They have provided srales , 
and are cpn^nulng to establish f^^unds for describing and evaluating concept* 
level Instructional iiiaterlals.>t, ^Wo&d, Richards, arid Merrill (1976) have 
developed and validated a measure^ of te^t-lnstructlon similarity with selected 
Constructs from the ISDP. ' ' 

This study deals specif Icallsf^ w;lth an assuQiptlon made In the ISDP that 
the 'test and generality stiould be cpnslstent. It coibpares performance of 
students given generalities that 'differ In three ways In the degree. to which 
they are consistent iirith the content "and behaviors requested by the test . ^ 
Items*" ; First,, a generality can present the student with content without' 
presenting the precise task behavlots^ he will be asked to perform. (This 
does not mean that no required behavior Is taught or implied, but tha*t the. 
speoiflc behavior required Is not taught.) This Is a low levet of consistency 
Sec^d, a generality can present the task conditions under which the student 
will be asked to perfonfi. Introducing separate generalities for each task 
making up a larger ta^k. We call this *Mlscrete rule** consistency, which 
requires that th^ mode of behavior be consi^ent (recall tested with recall, 
rute-uslng tested with rule-using^ tasks) * Thus, sl student Is worklng^lth' 
consistent discrete rules when an Item of information he .Is asked to learn ±fi 
taught and tested In recall^ioode or when a concept he Is asked to learn Is 
taught and tes^ted with rule-uslng*behavlors. If the Item Is taught with a. 
rule-using behavior and tested in a recall mode, the test and instruction are 
^consistent . r The third way in which we looked at generality-'test consistency 
involves task sequencing. This 1$ *Vonnected generality** consistencyt Gagn& 
(1970) and, Mechner (1967) discuss this level when they describe "behaviot 
chainsp" Attest that asks the student to perform sequential discrete tasks 
in a way which iSTiot presented in rule*-or practice form l^pcks- what maytbe 
an important consistency characteristic. 

Hypothesis . ^ 

Under the assumption that a generality is best that is. consistent with 
the required terminal performance^, we proposed the following hypothesis: 

Performahce on test items,^ toward instruction, cori- 

fidence_ln test item. .answers, ^nd total elapsed time -will be ^ 
Mgher for students experiencing connnected generality consistency - 
treatment than for students experiencing discrete rule consistency;, 
and these measures, for both the connected and discrete rule con- 
sistency treati?ents will be higher tfiSn for. the conterit-^nly fon-^ 
sistency group. ^ - * ' ' " ' "? ' 

Four separate hypotheses correspottiding to^^our dependent variables result 
fromjiJie above general statement. Each vill b"e treated separately in the 
reporting of resultst " 



Hethpd - . * 

, Subject'^tter Content ' ^ / ' 

An Introductory Statistics course at Brighaia Toqag University (BYU) 
198 selected as the experimentsl Situation for this study* The specific 
ntatt^r ^o£ hypothesis testing was chogen because generally lojw scores from 
past Achievement , tests indlcate^-^a^ high leve^ of 'difficxilty with the topic* 
The aAj«ct matter met the experimental specifications of having mul|:iple * 
ruii^ that couXd he taiigfht'^'as se^f^t&te^, disctete rules-or^as connected 
n4e^*^ ihe following hypothesis te^ts were eovered in the selected unit of 
iAB%tuciipn [Christensen^ 1974)! ■ , ^ - 

" ' ^ 1* Test for one mean when d is known* ^ « 

2* '-Test for one mean when o itf not known* ^ - ^ 
^ 3** Tesi for t«o means when the samples are independent* ^ 

> 4* Test for two^means when observations are paired* 

5t Teat for^ne proportion* 

6* Test for two proportions* * * .-\ , 

7/^ Chl-square test*^ ■ * 

„d* :Multinomlal test of hypothesis* ^ ' ^ 



Subjects 



y The subjects were 95 regular enrollees in a college statistics ^ 
undergraduate course* The course serves as 6ne of the choices for fulfilling 
A genetal education requireioent at BYU* Students received credit in the form 
of additional points toward tt\e^final course, grade for participating In^.tfie 
.2**hour session -and were informed that failure to participate would^^in effect 
penalise them, although the additional points were not dependent upon t^eir 
performance* 

Treatments 

The study consisted. of .three treatioents* Students in the first 
treatment group received a connected generality in the form of an algorithm 
which, presented both the content operation. and the task necessary to take* a 
studeAt from the reading" of the verbally stated problem-to the correct tfest 
of hypothesis and test statistics (see Figure, 6)* Subsequent practice pro^ 
vided two examples of each type of hypothesis test and test statistic* 
Correct answer feedback was provided on th^ reverse Side of^ the practice 
pages* '(See ?igure/7 for 1 sample practice question*) 
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Figure 6. Conit«ct«d-*rult algoritbft for selection of^ h7potl»ei« t«M:. 
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13« A new social ^tudies"^ program Is sui^posed to produce sigaifi{:an(ly 
better ^results than a program it is to replace* Students the 
course are matched ^ the basis of aex, XQ and G*P*il* and then the 
pairs are divided into **ney method^* and ^'old method** groups* Their 
scares on a final achievement test are taken asj evidence of' 
perf ormance * ■ ' * ^ ^ ^ * 

Qa* Select the Aumber for the appropriate test type from list A* ^ 
Qb* "^Select the number for the correct t^st statistic from list B* 



A-Type of Test 

1* test for one^mean, 
t distribution 



test for two means, 
dependent 



B-Test Statistic 
1. 



2. 



X - ^O X - o 



o- 



3. rest for one propor\ioi 




-r- I-' 
^ ii 

2. / 



3. //>- 



'■It ^ 



4. test 



proportions ^ 4. 



d - D d - I/' 



none of the above 



5. none of the^bove 



Figure 7< A sample practice question* 
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Students in the second treatment condition x^re presented the dis- 
crj^te (unconnected) multiple generalities used in determining the coi;j:^ct 
test o^ hypothesis and test statistic* These consisted of a- walk-through ^' 
of separate, very simple algorithms which took the student to the approp^riate 
test after the student*s initial decision as to the sEecific typiToT d'ata he 
was^worfci^g, with Xsee Figure 6)* Students in this grOup wre not given any 
directed strategy for connecting the^e behaviors or for usii\g them as part^ 
of ^n overall process to help them make their initial decisions as to the 
' nature of the statistical problem* Subsequent practice provided the student 
I with two samples of^jach t'ype of 'decision* Correct answer feedback wa^ ^ " " ^ 
provided on the reverse side of the practice ^ages* ^ ^ ^ 

The instructional materials for the third treafiqent condition con- ' $ 
sisted of the regular text used in the course apd directions for. providing 
^Appropriate, practice^ The practice directions consisted 6f a , sample item .of 
the^ype used in the posttest with instructions bp. practice the selected 
itein^found at the end of t'extbook sect^ons^. No' generality was provided for 

ig the type of test that a verbal .practice item may pose* The text ^ ' 
notv-help here, for, each test of .hypothesis was presented ih a. 

te lesson, \gave practice only in a stated kind of hy^thesis, and did 

not request students. tQ differentiate on the basis of kinds bt test* As all ' 
students had' previously been exposed to this material, this group became * 
essentially a control group* " . ' - , 

In <the previdTis ijtudy* CStudy^ 1 jjf/thls feport), it x^as hypothesized ^ 
that time would decrease as a^ resulx of ou;: treatments* In this situation, 
althou}^h it is desirable to reduce the amount- of time students take^on^n- 
Instruction^ it is expected that time wil\increase, based on Bloom^s^ (1^74) 
pbsorvation that quality instruction initially takes longer, especially if . 
the effect toward the instruction .has an'a student and confidence in mastery 
of the subject matter increases*^ 

V, Instrumentation ^ ^ 

A brief three-^questioh pretest w$*s administered to get some measure 
of student entry behavicx^* The pretest was * identical in form to the questions 
used in practice (Where giveif)^and to the posttest* Students were alsp ask^ 
to indicate lectins attended, materials read, and ^rkbook practice completed 
in regaiS^to the unit oft hypothesis testings 

,^asures for the four .dependent variables of interest in the study were 
provided for in the treatment materials and postt&st and*are clisctlssed below* 

Performance ^ The basic perfj^^nce't^Sk. required the student to 
-sel^'ct (1) the' appropriate hypjJtliesis *test"Tbi^ ^ problem -statement^^^^d (2). 
the appropriate statistical^ test associate^ wlttr the h>^pothes^s test (see 
Figure ^)* The 22 multiple^chbice^ -questions provided for 44 resp(jnses* How- 
ever, only the hypothesis test responses were used in the data analysis* 



**Bloom does not '(li'early define quality instruction except foj 'characterizing 
it as mastery learning*'* ^' * , 
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Senator Inocjng feels that^the bUls introduced by members of his party 
in Congress will be given a positive or negative ''TOrf^n the basis of party* 
^affiliation"* To test this he asse|scs*.the bills over a month's time, keep- 
ing track pf-the party affiliation of- those who took part in the voting 
(Democratic 'or Republican) andvhat the vite'was ("For*' or '*Against" or* 
**Abstain*')- ' ' ' " . 



Da. Select . the'number for^the appropriate test "type ffom list A* 
D b. Select thfe number for the correct test statistic from list B* 



^7- 



How confident are you in your answers tb-the above quellfbn? 



Vt/y 

Confident 



A^TypdofTesI 

1. test for hoinogeneity . 

2. test for twa means, dependent 
3* test- for two means. Independent 

4. ^est for tvp propoartions 

5. ^ none of the above 



BrTdst Statistic . 



2. 
3. 
4. 



^6 I . sy*^ 



5* none of ihe above 

■ 

* 

Do not returh^to this page once you have completed your answers 1 



I^iguxe, !)^* ^ X sample of the basic-, performance 
task^and confidence. rating* " * 



The test was designed to have more Items than inost students could ^ 
complete d.n the allotted time so that differences In time and number of - ^ ^ 
Items. completed for the separate treatment groups could be ascertalne<^ ' 

Affect * ..Questions at the end of the treatment materials and'^t 
the end of the posttest allowed students to respond to a five-category 
continuum of general^ affect l^Jerms of how well the Instructloiy provl*ded 
preparation for performance on a test*t 



t 



Confidence * A seven-point, semaptlc differential scale was Included 
after each of the 22r^'test Items In ordec to assess the amounttof self**per^ 
ceived confidence students had In their answers td^the multiple -choice 
questions (see Figure 9)* 

Time * All students were to'ioark the time from the walli clock In a 
space provided at (1) the point where they finished the first 11 Items on ' 
the posttest and*^ (2) at the end ^f the posttest-sesslon*- 

P^ocedure^ 

* Students were randomly assigned to one of the three treatment 
conditions* After the pretest, students were told there would be/three 
timed sessions, an^d they were requested not to begin any one q^^hem until 
asked to do so. They were also Informed that the materials ^ovlded would 
be collected before the test* .Finally, they were Informed that they would 
be pravlded with more Items In each sectloji than they wou^d jnost likely have 
time to finish and that 'they should work steadily but carefully* i 

The students were given 1/Zhopr for thQ study session* They were 
then r^qjaested to move on to the practice but were allowed to return to the 
study materials If 'they wished The practice sessldti lasted 40 minutes, after 
which the students recorded their ^ense of preparedness and at:tltude toward 
the mat^lals used* All loatetials were coMected* The tests were theo^passed 
out, and the'students were given 30 mlnut^^ to work the 22 ptoblem^* Tome was 
recorded on each test after the eleventh question and a^ain at the entl* of the 
test* Students respon4j|d to a second affective measure, and the materials 
were collected *.^ \ 

Design - ^ * , \ * 

The three treatment groups provided three levels of the main effect, 
level of generality consistency* A one-way analysis of variance fieslgn pro-^ 
vlded the statistical model for^both a univariate (AKOVA) and a multivariate. 
(MAMOVA) analysis of variance with two orthogonal contrasts for comparing means 
Xcontrol'vs* ^the other two groups for 1 d^ and connected rule vs* discrete 
rule group for I'df) * Each of the five dependent variables' was considered ^ 
simultaneously In 'a MANOVA to correct for Type I error* ANOVA fesults ; 
on single dependent variables were the^ Interpreted If the exact F-ratlos ^ * 
from the KAJNOVA contrasts warranted fui^ther consideration* A geneirallzed ^ 
analysis of variance computer program which adjusted for an unbalanped design 
was used (Bryce & Carter, 1974)*' , , ' ^ 



Resul'ts ■ 



* *f 



^ .^ Jiypotheslg I ' * . 

^ " ^Pe:l:formahcfe on test^T^teBis for. the connected rule consistency * 
; * araup'wlli be/hlgber than. that for tbeMlscrete rule , consistency 
R|oup, and perforbance for botb_ groups will be higher than the 
' * content , only Ot'ow consistency) group. / " ; 

. ' ; ' \ - * 0 ! ^ 

Means and standard deviations on performance scores for the three treat- 
ment groups are fo*und in Table 3. A univariate analysis of variance and an 
orthogonal co^patlson of means (Coptrol vs. Connected and Discrete, 1 df ; 
Cbnnepted;^s. piscret^iliu^/ 1 df) indica\:ed a significant difference 
be^ueen both the connect^ 'i:bij.e and discrete rule consistency groups as 
compared tp^the control gi^p^ F (2^W) - 4.21^ £ < .05. There was no 



significatit ' difference between the connected rule and discrete jrule^con- 
sistency groups.' 



Table 3 



V-^^ ' Hearts and Standard Deviations for 
f; * Depeqd^^ Vwiables by Treatment Group 



Dependent' 
Variable ^ 



, (^nhected 
^Rule ^ ' (N=33) 
^ean^H S.D. . 



treatment Group 
D^crete 
Rule . (N»32) 
■liean , S.D. 



Contror (N»30) 
Mean' S*D/ 



Performanq^^:^^jS*^ ' 




.54 


10.54 


.54 


9.10 


.56 


Affect Toward 
Instruction' 


^_ . 3-.06. 


.06 


3.38 


.07 


2 ; 92 


.07 


Cohfldence ln'?'\|\- 
Arigwejr^ to 'p"*! - 
T^t Items f Vi-i 

Time oo* 
Pidi^ttest 


■ 3.79 
7 

• * ■ 24.39 
-ii. : 


.31 


4.19 
* ?3.84 ■ 


- .23 
.32 


3.60 
23.9 


.24 
.33 




, ' ilj^orhesis 

^ Affect tpuard insCiructlpri will be higher for the connected 
rvfXe^ consistency group than for the discrete rule consistency ^ 
/grotip,^and"aff€;iCt toir botJt groups will be higher than for the 



' content only (low ci)ns;lste^cy) grouj). 



. Cleans and standard deviations on affect toward Instruction after the _ 
treattnent^ondltlon are found In^Table 3« A univariate analysis of ' 
varla&ce and an orthogonal comparison of means Indicated; that all three 
groups were significantly different from each other with the discrete rule 
consistepcy group highest^ the. connected rule consistency group neict highest^ 
and content only group lowest In affect;, ? (2»92) * ll«2l» p < «05« 

Hypothesis 3 , 

Confidence In answers to test' Items will be higher for the 
comlected rule consistency group than for the discrete rule 
consistency groups and both groups will be higher in confidence 
than the content only (low'conslstency) group« 

Means ^nd standard deviations for confidence In answers to test Items 
are foundsn Table 3. A unl't^arlate analysis of variance and an orthogonal 
comparison of means Indlcated^^ioyslgnlf leant difference between any of the 
three tr^tment groups: £ (2^92) » 1.60^ p > .05. 

Hypothesis 4 ^ 

Time to complete rule using polttest items will be longer ^ ' 
for the connected rule consistency group than for the discrete 
' rule consistency groups and both groups will take longer than ; 
the content .only group. 

Means and standard devlatlons'^for time to complete the posttest^are foi^ 
in Table 3« A unjLvarlate analysis of variance and an orthogonal comparison 
of means Indicated no significant difference between any of the three treat- - 
ment groups; P t2>92)> .gi^ £ > .05. ^ ' 

Pis CUSS lonr \ . 



^ The study 'investigated the extent of the h^ed for test Items* to be^conr 
slstent with their generalities In content representation and In' task behaviors 
on both a discrete and connected rule level« The re'sults Indicated that 
students who learned from materials that were conslst^t^nt only on' the content 
representation level had significantly, lower scores and affect th^n did students 
whose InstructlQn also ^was consistent With the test. Item on the task behavior 
level« Neither confidence nor time was significantly d^^ffexent across tr&at- 
ments« / * ^ 

The specific constraints of t^f^study may have obscured greater differences^ 
especially the hypothesized differences between the discrete ru^e and connected 
rule consistency ^oups. The time we could arrange demanded that we run the 
entlre^study (from Introduction to Instruction to practljce to test) In one 
2-hour dltting« Thus» the students In the connected'rule' consistency groups 
wtio had the most new in^terlals to learn^ had very little^ time to encode the^ 
rather lengthy algorithm. « Given more tlme^<we could have explicitly taught^ 
them the three or four major steps Involved In the algorithm before presenting 
them with, all the detail and» thus» allow fox easlejr chunking (Killer^ 1956) of 
th% Aaterlala« As It was/ the discrete and connected rule 1::onslsteScy groups 
ma y have Responded more in a forward processing manner (Mayer » 1975b) In which 



chey did as well as they did base<J upon the expectations aroused, not only 
by Che Initial statement of the terminal beliavlor but also by the on-going 
practice* Familiarity also possibly "impacted upon the^'results. The students 
had worked with materials similar to that provided to the discrete r^ile con- 
sistency sroup, while the algoVlthmlc approach was not a tool familiar to ^ 
the course. A study that allows. adequate time for encoding of the materials, 
preferably run with several meetings of the groups, should be made^ 

' Despl^te the const;ralnts of^ the study, the pres^ce of test^generallty 
consistency beyond a simple coptent-^nly level clearly resulted In better 
test performance and affect. Teachers and developers would do well to give 
practice In the specific behaylors required by a terminal task. 
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STtJDY 3: VA|jiriATION, OF THE INSTHirCTIONAt STRATEGY 
' DIAGNOSTIC PROFILE IN PHYSICS 100^ 

^ Overvlev and Hypotheaea , , - * ^ * 

The prea'i^t atudy a^aumed that already designed ^nd Individualized 
loaterlala on concept leyel taska could be further upgraded by an example^ ^ 
practlce-feeilback aequence for each generality, In accordance vlth the^ 
hypotheaea -^atated In Merrill and Wood (1975). -Th^'ISDP alao aupporta the 
generality accepted principle that teat Items for^ claaalficatlon ot rule- 
using taakfi ahould conalat of unencountered Instancea., Much Inatructlon 
Ignorea thla dictum. Thua, thla atudy analyzed the teat queatlon type for 
the material used in order to compare atudent. performance pn l>oth encountered 
ajld unencountered Instance Items and aacertain' dlf ferencea In atudent per- 
formance' when the study materlala are upgraded. 

^ ■ ■ 

one hypptfaeaea were baaed on the aaaumptlo^ that an Increaae In the degree 
to which -tea ta and- Instruction follows the prlhclplea preacrlbed In the ISDP 
reaulta In higher acorea on teata with unencountered Inatance Items, and com" 
parable a cores on teata ylth prevloualy encountered Instance Items. 

^ Mgthoda 

Subject Matter Content v 

The aubject matter conalsted of six units (comp^laing the aecond * > 
quarter of the courae) ^r an Introductory physlca courae at Brlgham Young 
* Unlveralty. There were aeveral reaaona for aelectlng thla subject matter; 

1. The courae met the criteria that It be conde|>tua}._ly baaed, a'^^ 
quality -the dealgnera and Inatructora of the, courae ^de^red. 

2. The courae, and eapeclally the atudent atudy guide, waa already 
carefully dealgned and yet ahowed deficiency- In one or more of the areaa 
zne&aured by the ISDP. Too often researcher^ and theoretlclana h^ve been 
accuaed of shooting down atraw men aa they compare materlala they had developed 

^ in an hypothealzed better way a^alnat haphazardly preaented "undealgned" leasons. 

. 

3. The materlala covered a fairly broad range of }:oplca. It la often 
aljoplet to create dlf ferencea.. In a brief one-ahot aegment of material* Under 
such a'conditlon, the novelty of the appiroach— and Ita brevity— go hand-ln-hand 
to ^en^rate unusually attong attention to the taak. To get'^t r^ea^ dlfferencea. 
It aeema jieceaaary to have materlala used over time. 

4. There la a real and aubatantlal challenge to ahow dlfferencea Itr 
the leaaneat and varied woi^d of the on-going claaa rather than in the laolated 
laboratory aettlng (Glaaer & Resftlck, 1972). Our job la to ahow that the 
efforts ifhich go into nuiterlata dealgned apcMdlngr to atated hypotheaea reault 
in real-life- dlfferencea. ^ 

^ : , : 

^St'udy conducted by K. D, Merrlll,_^E. _.V. Schmidt, and R. F\ Norton. 
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5*' The coui^se was an introductory course that serves as pne of 
several options ta fulfill a general education requirement, ^us, the 
students representeiha broad spectrum of co^llege undergraduates with a 
diversity of interests and skills* " ' . ' ' . 

6* Data on students enrolled in tKe course indidbte that, dueling ^ i 
^the Fall Semester of 1975, tw-thirds of the students eitiier withdrew. 
unofficially from the pourse or received a grade of incomplete* Student- 
paci^ problems obviously contributed ' to^ these results, but problems no doubt 
existed with the tests and instruction as^^well* 

The material selected covered the following topics: 

1* Motion and Forces* . ' ^ 

^ 2* Forces ^n Fluids at Rest* 

i 

3* Pressure in Moving Fluids* * - * , t 

A, Conservation of En*ergy* ^ \ ■ ' 

^ ^' ^ ^ - 

, 5; Kinetic Theory of Matter* 

6* Law of Increasing Entropy* 

Subjects ' ' * r 

The st^bjects were A3 students from two sectJLons of a summer session 
of the above-mentioned course, which fulfilled part of the general education 
requirement in the physical sciences at the university* Other subjects, 
representing repeating students and students who were not present during the 
initial phase of the study, were too few in number within their groups to 
analyze meaningfuXlyt 

Treatments ^ , 

\ 

The two treatments were; (1> the regularly constituted 'class study 
guide and (2) a study guide whose generalities were reinfx^rced with example- 
practice-feedback segments according to ISDP principles* ^Moreover, eight 
imencountere^ ^stance item que&tions were added to the regular seven-item 
test, which ^nsisted entirely of encountered instance or generality items* 
Each questixin^^d several parts to it, and there were four vetsioi^s of the 
test* 

Procedure - > ^ ^ ^ ^ 

A preliminary study of the nature of the test question and the cor- 
responding s.tudent ^erfonnance (see^ the appendix) was ^0 in .order- to ascertain 
which^tests could be upgraded thj?Ough eliminating previously encountered in-* 
stance iteps and adding unencountered instance items*^ This 'evaluation also 
allowed selection of a -unit on whicK students sliowed problems in test, per- 
formance* Following this, both tKe tests ^nd material were upgraded according 
to the principles of the ISDP* ^ * ." 



dePcriptionnOf this study, which was conducted by M* Dt Merrill^*^ 

the appendix* ^ - 

47 ^ * ' ■ 



Norton,, and R. V. 'Schmidt^ Is provided In Che appendix* 
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Students attending class the first week were randomly assigned to 
the treatments. They were requested not ££_atudy, with or share their 
materials with anyone whose materials did not^match theirs. The visual 
difference in materials was immediately apparent. In this on^going situa* 
tion, studying togethej: had to be allowed* The randomization should have 
talcen care of any students who might have collaborated using different 
materials. This was checked later through a questiomiairet two students' 
indicated that they had looked briefly at or studied with the treatment 
materials they were not assigned. * 

Students . could take the IS^item short answer ^essay test at a testing 

center at their own convenience* The e^erlmenters picked up the test from 
the regular graders on a dally basis, regarded them blindly, and returne^T 
them, the next day to the testing center for distribution, keeping copies of 
each exam for further reference. 

' Students not in attendance the first week (N 6) and students retaking 
the course (N ** 7) also took the test, but their ninnbers were insufficient to 
allow an analysj^ of their performances* 

The amount of time required far taking the test .was also recorded, and 
an affective questionnaire was administered after the completion of this phase 
of the course to. see if there were any general differences between the two 
groups. 

Design 

4 

The design was a post-test^nly design with subjects nested in materials 
but crossed with item type* Two levels of the main effect ("regular" and 
"upgraded" materials) and four dependent variables (scores on encountered 
instance items, scopes on unenco^ntered instance items, time on test, and 
affect) were considered* A two-way analysis of varriance design prodded the 
statistical model for a univarlable analysis of variance to test subject per*- 
formance* Rummage, a generalized analysis of variance computer program to 
' handle an unbalanced design'and adjust for other effects was used in the 
analysis. A t-test was used to analyze time data, as not all ^sts carried 
this infonnation* 

Results . " ^ 

Hypothesis 1 

Tests requiring classification or rule-using behaviors for 
unencountered instance items result in lower scores than when 
the itema consist of previously encountered instances* 

Meana^and standard deviations on performance scores for the two treatment 
*^ groups are found in Table 4* A univariate analysis of variance indicated a^ 

aignlficiant diffe'renpe between previously encountered instsnce items and^ 
jjj^^ * unencountered instance items* in the hypothesized" direction: F (1>41) "5*5, 
P < -05* , , 
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Hypothesis 2 . ' 

Unencountered instance *items requiring classification on 
rule^u^irig behavior result in higher scores when' the 'degree 
to which the instruction follows ISDP principles is increased 
.aver instruction which does not g:enerally follow ISDP principles* 

. \ . ' ^ ^ 

Means and standard deviations on performance scores for the two treat- 
ment p^roups^ are found in Table '4* A univariate analysis of variance in-^^ 
dicated no significant difference between the two groups; £ (1,41) « *29, 
£ > •OS* 

' \ Hypothesis 3 

Time to complete a rule-fusing .or classification test is 
greater when the degree to which instruction follows the ISDPo ^ 
is increased over instruction which does not generally follow 
^ ISDP principles, ' \ • ^ / ^ - - . 

' As the testing place and tiine wa3 out of our hands^ time data w^s made 
available for only 17 of the subjects* A^t-test nin on the available da^ta 
(Mean of 87/S»D» of *36-'f^r the Upgraded group and Mean af ^l^. a*D», of 24 
for the Regular group) indicated no significance; ^(17) 1*116, £ > ,05* 

DigcussJton , , - ' ' * ' ^ ■ 

The s^tudy attempted to ascertain if one could improve student performanc 
on a physics rest by upgradidfe bis syllabus — mentione^d earlier as the major 
teaching dev4,ce — by adhering to^SDP principles* Because we intervened in a 
pn-goin;; class^ this ^lad to be' attempted without controls on the teacher^^G 
lectures^ videotaped helpsp or the text* Though th^ differences noted were 
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in the hypothesized direttion (se^eYFable 5), they Wre not significant. The 
large standard deviations indicate* that we had not captured a substantial 
source of variability, probably-due to course utaterials and inforntation 
b^ond the syllabus. Contrary to indic^ations on previous course pjarticip^- 
tion, gleaned from several former physica students and ain instructor of the 
course, a survey we ran indicated that all the students ledio angered the 
questionnaire (N " 28) attended virtually all the class 'lecturea and used 
the text for each unit of material. \ \ ^ 



\ 



Table J5 
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Mean percentage of Items Answeted Correctlj^ for 
Students with Upgraded and Regular Materiajj^ 



\ 





Upgraded Materials 
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65 \ • 
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Unencoiintered Items 


69 


74 
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fforeover, the "regular" materials were; as mentioned, already rather 
carefully developed. Although they did not support each stated generality 
directly and consistently with examples and practice, bot^ examples and 
practice were available to the student who hunted for them. Thus, since we 
only upgraded the generalities present in the original sylldbus (as a promlae 
not to change the course for one group of students), we probably were too \ 
optimistic in the resultd we thought it would create. That upgrading from \ 
regular-class, *'non-^developed*V materials does create highly significant dif- 
ferences has recently been demonstrated in a study comparing the results of 
students taught by ISDF dnd "regular" methods in nutrition classes (Richards, 
Richards, & Merrill, in press). 

Our dependent measure also' had constraints plaped^ upon it which rendered . 
lt.^es^ sensitive than it could bei The instructor felt that, to kefep the 
initial contract with his students, we had to keep the original sev^n questions 
on which the students would be graded. In order to obscure which questions 
these were, we had to write ours in the same essay format. Al^o, we were 
allowed only to double the length of the test,^ so we could* not go beyond . 
eight additional items. This was not sufficient to test all generalities at 
least twice, especially since we were bound to the essay form. Since greatest 
majority of atudent test items consisted of encountered instance items, and 
our, questions . consisted of unencountered instan(^e items, this ,disflnction was 
easy to treak out. ' \ ' 



Thdugh time was not a aignlf leant effect'^ iit would be, perhapa, given a 
aampling of all the students. 



The sample In the affective questionnaire , was too small to use for 
drawing any valid conclusion^: There wad generally mlx^d response from 
,bo;h groups^ with conmtjentSt when made» ihdlcatltig sbsie dissatisfaction 
with the length of .t|ie upgraded materials over what they were used to but 
a greater security in the -subsequent test answers and a desire to go to 
the syllabus. for answers rather than mqylng from the syllabus to the text 
'as some previous students indicated thWy did* 



This research helped establish several guidelines for further^ Inter- 
vention studies of thl^ type* Flrst» thej experimenters shpuld' have control 
over all the Instruction^ Including lecture materlarlirThey cannot assume 
that general nonpar tlclpat Ion at lecturesjln the^past will be the'case In 
' ^e present. Secon41y» the experlmen^rs' must have full control of the test 
and tedtlng situation. This will allow a satisfactory and sensitive measure 
of student performance on the generalities taught as well as make possible 
complete data on time and affect. Following these guldellnBs^ we can then 
^ perhaps test the power of 4:he ISDF against materials developed at a level , 
" similar to that of the physic^ materials and to test /isbP developed materials 
over time within the framework of an o^-golng class. The rationale for 
selecting this type and amount of subject matter content^ as dlscusse^d In 
the Methods sectiona ls important ^o consider In conducting stuqles on the 
effect of Instructional iDoaterlals on student performance. 

\ 
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- ^ CONCLUSIONS i . 

The research reviewed lh^l6at^ that the propositions underl/lng the ISD? 
Trctf lie seem to be valld« Vhlle the data reported lir this document Is some^ 
what Incopcluslve and not suff^lent to make Unqualified statements. It Is, ^ " 
nevertheless, -positive. When cons^ered with other data on the ISDP (e.g., ' 
WoQ^» Richards, & Herrllii 1976),. It seems reasonable to assume that, when 
the ISDP Profile Is used as a guide to analyze and modify existing Instruction, 
the resulting performance of students Is likely to be more effective. This * . 
Is especially likely when' the tests as well as the main line Instruction, can , 
be modified. It is less likely when only the student syllabus is modif led« 
The ISDP deeg eeem to have^ considerable potential as an instructional evalua- 
tion and envelopment tool. * ^ ' ^ 



^ » , RECOMMEHDATIONS " 

1. th^ISDP, as presented in the ISDP training manual, is^ recommended' 
toj;^^^.hy JSlavy iiiBtTixctix>nalA^elopeTa and evaluAtors; However,- it should 
be ^&pnsid«red an experimental' too^ and should be used only by experienced 
HtstTUci^j^l t^chiioIjD gists who* can appropriately adapt its use to^ various 
aettinga.Han^ clrcumst^fnces. . * ' \ " 

2, The present effort has increased cAir understanding ^f the^ ISDP and 
has considerably increased our'abilijahto diagno'se and prescribe modifica^ 
tions in existing ;lnstructional matenala which result, in imptoved student ^ 
performance. However, our understanding of the instructional diagnosis and 
preacription process haa merely scratched the surface. Because of its 
apparent us*efulnessT it^is recommended that. ISDP validation and development 
efforts continue so that this instrument \can become dn easy to use tool for 
all instructional developo^ent'^and evaluation pjersoimel; 
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Introduction 

. The Physics Department at Brlgham Yopng Ufaiverslty requested that the \ 
:Div4^sion ^o£ Instructional Research, Development and Evaluation mate recom- 
mendations fot inq)roving the. t^aslc physics course at the University*- Dtfiplng ■ 
' tiie' Fall Semester of 1975, twa*thirds of the students eitiier withdrew un^ 
^officially ^from" the course or received a grade of /incomplete!* Student pkbing. 
^jprot^lems 6bvib,tis3:;y ^Contributed to these results, ^ut problems noMoubt existed 
^iilth the* tests and instruction as veil* ' . . 

^ Previous valuation of the PhysicslH-QO course at B7U has demonstrated 
vthat ithe course, intended to teach conceptual matter content, contained 
material that -was often deficient in the rule^eicample^-practice proposition ■ 
of the Jiistructiooal- Strategy Dlagaost'ic Profile. . ^ - 



This study examined ^the conceptual correspondence of the test; items to ^ 
the test prescriptloiid of the ISDP*' Student. performance ^s compared on the 
various types^ of ' iteins that were incl'uded on the tests* 

Method . . ' * ■ 

^-T ■ ' - ; ' ^ . ^ 

The' Physics D^^rtment provided the pool 'of test' items from which all of 
the tests administered to the students were constructed* Each of the test 
itenui'was classif;led into one of five categories according to the type of 
coij^ent they measured: 

^ 1* l}nepc0Qirtered iriquisltory instances (leg)— ^for questions lii which 
. the stud^ft ^as aalke^ to apply a rule (given or not giyen) to a particular 
ttistanpenot previously. encountered; 

2* FattlaHy .encountered in^qui'sitory rlnstance-rrfor the same type of 
tjuestlons as ^n nmnber oae ^bove. But w^t^ the particular instance had been 
only partially ejicQ^untered before* - ' - ■ ^ ^ 

^3. V^^^u^^^^^^ in<|uisit6ry ihsta^tce — for the same type o£ questions- as 
£a nuji^'rs one and .two above, but 'where .the^ particular Instance had been 
prevloualy encoun^erefd in the in^ructiohal materials; \ ^' ' 

. K y 

4. Inqulsltory generality /IG), — for question^' in which the student was 
asked to r^iiteitt^er pr recognize^ a rule^ statement o^ concept definition* 

S; Ml^cellaneptfS category (M) — for qi^estions where the student was asked: 
(a) to vcit* Evidence <data tfr logic) for a given proposition* (S) to give or • 
recognise' fuperbrdinate, coordinate, or aubordinate 'relationships among gr^ 
betweeh prqpositiott^ or concppta, or (c) to'Vemember a given conatant or some 
•peci'fic piece of data, a fact, etc*, which la an identit^y* 

' , ■ * < 

■ Itofft^ the teat? itemS; contained- more than^one category, of question within 
ttve i'tj^*'^If any questions occurred within ^ l^em, the whole item w4s. 
'claMi^fled leg* If an IG qiiestioawaa. combined with a M queatipn, - the whole 
item^vas Jcla8slfied''l{*^^Interrater reliability yas atrengthened bjr having 
both ratet« ritfte the same Aems separatelxAnd then compare the, reaiilts* The 
fey dlsAgrcf^ments vete Siacussed until consehsug was reached on all items* i 



An item was classi£i6d as encountered:' if the answers to tvo-thirdk or 
more of the questions constituting the item were foun4 anywhere in the t^t, 
the syllabus, or the television lectures. An item was classified a& une^^ 
countered if^ one-third or fewer of the questions constituting the item were 
encountered in the above mentioned sources. Items falling in between thes^ 
two cutoff points or items where a similar but not identical instance was 
encountered in the lesson materials weVe qlassified as partially encountered. 

Independent Variables ^. ^ " \ 

An analysis of variance was run, using three Independent variables: 
C?) the three examinations over three different ,ar^as of subject matter, (2) 
the seven test items used on e^ch test, an;} (3) the five c^ategories indicating- 
the content t3rpe of each Item. 



The first examination co^^red the^trst six chapters of the text and ^ ^ 
aimed at a conceptual understanding of lIe*on^s first two laws of motion. .The 
second examination covereid chapters seven through teh of the text and aimed at 
a conceptual understanding of th^ laws of force and iDotion, conservation of 
energy, the kinetic theory matter, and the law of entropy^; The third 
^examination covered chapters 11 through 14 and aimed at a concep4:ua;L under-, 
standing^of the properties of waves, electricity, and magnetism. 

The test item humber was included as an independent variable because 
it served as an index of the difficulty 'level of the various items. The sixth* 
and ^^eventh items on each test (A level items) were designed by the developers 
of the test to be'more difficult than ^he fourth and fifth items level 
items), and these in turn were designed^ to be more difficult than items one, 
two, and three (C 'level items). An Inclusion of this variable in the analysis^ 
of variance enabled an empirical evaluation of the preassessed difficulty 
levels of the ^t ems. , ■ ^ t . . ^ 

^ The five content t3rpe categories were Included to Assess which types 
of questions were being answeiTed^ most effectively by the students. 

Dependent Variables , - ' ■ 

The Physics Department ha^^alre^dy gathered^data on the number of - ^ ^ 
students that had missed ^acK item in £fte pool of test it'ems. The number of^- 
times each item was used on a test was calculable from loiowing the total., number 
of tests given and the procej^ure ysed to gpherate'the various testd that were 
used. From^the above information^ ' tW'percentage of students answering e^ch 
^test item correctlV could be i^etermlned. This percentage was used as the 
dependent variable in the analysis of variance reported in the results section 
of this paper". * 

i The final data analysis design was a\,3 x 7 x"5 matrix that can most 
i:l,*arly be understood by looking at the' design diagram in Figure A-1. 
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Figure A-l* Design for evaluation of test item 
\ . type and student performance* 
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Results - ^ 

^ Table A-1 shows the mean perpentag^ of student^answerlng Items cor-*' 
rectly on each of the three examinations, the mean percentages of ^^tudents 
^swerlng questions correctly under each t^st. item number, and the means 
formed, by th^ examination and item number interactions* An analysis of * 
variance shoVed that the differences among examinations means were signi- 
ficant (F. » 16*366, £ ■< .01) aa were differences among ican number means 
^ 23.747, £ < :oi). ^ ^ "^^ 



Table A-t 



Examination vs. Item Humber Matrix Mean PBTjiL^ata^fij- 



of Students^ Answering^ Items ^Qorrec tly 



Item > 

Humber 




1 '"^ 


Examination 

2 


*- 


Heans 


1 




' 87.945 


83.141 . 


82.998 


84.816- 


2 




86.447 


80.1% 


•.?5.807 


81.250 


3 




72.444 


82.376 


72.177 . 


75.418 


4 




79.284 


71.663 


61.232 


71.590 


' 5 > 




73:763 


63.J85 


671.017 . 


68.745 






7i,300 


60.378 


■ 46.421j 
56.509 


58.372 


7 




\. 58.67? 


^0.142 


58. 182 


Means 




. 76.021 


73.2^1 


65.811 ' 





Table A-2 sitows the mean percentages of students answering each type of 
test question correctly, the mean percentages of students answering questions 
correctly under each test item number, and the means formed by th^ content 
type and item number Interactions. • ' 



Table A-3 shows the mean percentages of students answering items correctly^ 
on each of the three examinations, the mean percentages of students anawerlng 
each type of test question correctly, and the means formed by |examinat ion an^i^ 
content type Interactlchsi An analysis of variance showed that^^ the differences 
among the content-type means (66.0777^72.505 , 73.388 , 73 * 024,* and 747159) were 
also significant (F - 2.^49, £< .05). . . * * 

The nattire of 'these differences wa^ .analyzed using prediction coefficients - 
and is reported in the discussion section which follows, table A-4 glVes the 
percentages of items used from each content type on each examination. 



Table A-2 ' 

* 

^ Content* Type vs* Item Number Matrix Mean Percentage 
*o£ Studejits Answering Items Correctl^y 



Item / 
Number 



Question Type 



Unencpun- 
t^red leg 



Partially 
Encounr* 
tered leg 



m 



'd'leg 



IG 



M 



Means ' 



3 . 

5 , 
6 

.7 ' 
Means 



59,209 
61,622< 
64.186 
'58,785'^ 
•64, Us 

66,t)77 



\ 86,^^7 



79,713 
70,549 
92.009 
. 45,253 
53,574 

. 7^,505 



87^556 

74,180 
77.684 

* 

70,486 
66,425 
59,166 

.73.388 



80,146 



^7,611 
78 ,'7 96 
72,476 
63,813 
41.333 
65.864 



, 87,273 

» 



79,333 
81,22P 
72,666 
0 

76,250 
48.610 



/3,024 74,159 



84.816 



^81i 2150 
75,41« 
7i,590 
68,-745 
58,372' 
5?, 182 



Table A-3 . - 

Examination vs. Content Xype^ Mean Percentage of Students 
^ Answering Correctly Items, of Each^Questibn "Type 



\ 



Question Type 



Examination , 



, 2. 



^Heans 



Unencouat«red leg 71,876 

Partially. Countered lag 78,058 

Encountered leg ' 78,084 

K " 78,200 

M ^ 73,54^4 

Me^r? - . 76.021 



64,144. 
73,504 
75.238 
73,470 
78.495 

73,291 



V 



61,568 
59,735 
66,842. 
68,204 
70,743 

65.811 



66,077 
72,505 
73,388 
73,024 
74,159 



The Percentages of Items Used' from Each Question 
Type on Each Examination 







Examination ' 




'Question Type 7 


, 1 ^ 




3- J 


Unencouiitered leg ' * 


3U 


20% 


- U% • 


Partially Encountered leg ' 


19% 


7% 


■5%' 


-Encffurtteired leg 


T 25%, 


35% 


46% . 




—10% , - 


.34? . ' 


' 29% 


M • 


15% - 




9% ' 




100%. 


100% 


100% 



Discussion 

, The difference among examinations Indicates that significantly fewer 
students responded_corredtly to the It^ms on^ the third escam (see Table A^l), 
It Is possible that' the Items were more dlfilcult or that, because of the 
end of the samester, fewer students retook the third exam than ^retoolc the 
first and second exams* \£ach tliBe a studenf retook an examlnationV ev'en 
though Che Items w^rjg. different than on the pifevious exam, he was llleely to 
do better than he did the time before j>ecause of more stpdy lti_^e aifea :phere 
he was deficient. This would mean that the average percentage pf students 
answering Items correctly was artificially elevated for both the first and * 
second examsr^-more so for the first than for the second. Regardless *of the 
question type. Items on the third exam were^mlssed more ofteii than the cor- 
responding type of Items on the other two exams (see Table -A'3)« 

The sixth and seventh test questions on each exam were consistently more 
difficult than ail of the other questions (see T^le A*l)* However^ question 
six on exam one was not significantly more difficult than questions i&tiree and 
five. The overall means for the seven question numbets- Indicate that questions 
four and five fell In the middle range of difficulty as Intended, but -thls^'was 
not consistent when th^ three exams were considered separately. 

Although questions six and seven were more difficult than the oth^ts^ they 
were not measuring a. higher lev^l of conceptual understanding, asjnlght be^. \ 
hoped, but, rather more obscure details encountered In. th^ test, sxHabus*. or 
videotapes* It might b,e more meaningful to use previously^ unencoun^ered ' . 
questions as A and B level ltems« This would tend to award ^s'and AsVon the 
basis of a better conceptual understanding of^the material rather than on the 
ba'^ls of ability to remember more Qbscure detail. \ ^ ' 



^Regardless of the type of question Involved, itemB six and seven ^ere 
consistently missed more frequently than the other item (see Table A-2)* 
This is likely ^ reflection of the tendency for A^level items to deal with 
obscur-^ details** It is also interesting that- on the unencountered inquisitory 
Instance queAlons (unencountered leg) ^or the B and C levels, each of the 
ittean percentages falls below the grand mean for its respective question number* 
This ^s as we vould expect for more difficult questions* Yet l^or the A level 
Questions, the unencountered leg questions have mean percentages equal to or 
higher than the grand means for their respective question numbers* This may 
mean that the students have acquired a set response to unencountered unobscure 
items versus unencountered obscure items* For example^ 'they may be skipping 
the unencountered unobscure items without spending much time on them because 
they realise that they have never seen them before* At the same, time, be- 
cause the A level items involve more obsqure material'^ they are spend^g more 



time to thfnE"an3 are cdihlrig^pjiath ^mofe right "an^Sers pii^eReir ownr 

^ 

The unencountered inst^ce questions are significantly more difficult 
< *05) than all other test question type? as we sfxght expect if they were 
measuring understanding at conceptual level rather tnan at jus^ a memory level 
(see Table A*3)* The partially eTifdountered instance questions^ the encountered 
instance questions^ the Inquisitory generality questions, andlthe miscellaneous 
questions all seem to be a^.the same level^of difficulty for tVe students* How- 
ever^ the partially encountered instance items on test three ans slightly 
'(though not significantly) more difficult than tTi^unencounterep instance items* 
The partially encountered items in tests one and tvi»>^but more l^imllar to the 
unencountered items In test three^. 

The percentages of unencountered instance questions on the variaus exams 
are also very^ interesting* Because the;^ are the most difficult items^ one 
mlg^t have expected a poaitive correlation betw^^ t^e. percentages o;f such, 
items and the performance by the att^^ts orTl^he^ tefi^ts* However^ there >;a's a 
negative correlation (see T^ble A^A)'. Although test ^hree was;the.,most 
difficult for the students; it ha^ only 11 percenjt' of the njost (};tfflcuit 
question type. This tolght mean that the suBject matteif' tested -in te6t ; 
three was inherently more- difficult or that the Instruction- in this at^ea 
was weaker. ^ - . * ■ ' 

- . ' ■ - ' . . / 

The present study will be expanded tf se^ if student performance on upen^ 

countered instance questions could be loljproved by following the princples of 

effective instruction recommended in the IiiStrUctioniii Strategy Diagnostic' 

Profile. V ' . 
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